Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge opaque closure modules with the rest of the workqueue #50724

Merged
merged 2 commits into from
Jul 31, 2023

Conversation

pchintalapudi
Copy link
Member

This sticks the compiled opaque closure module into the compiled_functions list of modules that we have compiled for the particular jl_codegen_params_t. We probably should manage that vector in codegen_params, since it lets us see if a particular codeinst has already been compiled but not yet emitted.

@pchintalapudi pchintalapudi added compiler:codegen Generation of LLVM IR and native code compiler:llvm For issues that relate to LLVM labels Jul 29, 2023
Copy link
Sponsor Member

@vtjnash vtjnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. Though I hope in the future we can make this workqueue a multi-threaded-supporting channel (once the JIT stops making bad assumptions about needing to optimize in the same order as linking), which is why it wasn't threaded through like this before.

@pchintalapudi
Copy link
Member Author

Maybe codegen_params can hang onto a reference to (workqueue, compiled_functions, external_functions) and have that be provided by JIT/AOT? That way we should be able to deduplicate both across the JIT and across whatever code is being AOT compiled.

@pchintalapudi pchintalapudi merged commit 441fcb1 into master Jul 31, 2023
6 of 8 checks passed
@pchintalapudi pchintalapudi deleted the pc/oc-workqueue branch July 31, 2023 17:58
@vtjnash
Copy link
Sponsor Member

vtjnash commented Jul 31, 2023

I think each of those are largely an implementation artifact of not being properly isolated from the enclosing scope, rather than an actual proper feature. In a future implementation, those should all be removable since each compilation unit would be self-contained + a list of other units that are required. It would then be up to the system, post-linking, to ensure all of those dependencies are satisfied. Right now all of that happens before compilation, which limits threading scalability of compilation.

@pchintalapudi pchintalapudi added the backport 1.10 Change should be backported to the 1.10 release label Aug 2, 2023
@pchintalapudi
Copy link
Member Author

Marking for backport to fix the issues noted by Keno earlier.

@KristofferC
Copy link
Sponsor Member

@pchintalapudi, the open PRs from you with backport label tend to have a lot of conflicts that I cannot resolve myself. You could take on the effort of cherry picking these on top of #50708? You could just push them there when its done.

pchintalapudi added a commit that referenced this pull request Aug 10, 2023
This sticks the compiled opaque closure module into the
`compiled_functions` list of modules that we have compiled for the
particular `jl_codegen_params_t`. We probably should manage that vector
in codegen_params, since it lets us see if a particular codeinst has
already been compiled but not yet emitted.

(cherry picked from commit 441fcb1)
KristofferC added a commit that referenced this pull request Aug 16, 2023
Backported PRs:
- [x] #50637 <!-- Remove SparseArrays legacy code -->
- [x] #50665 <!-- print `@time` msg into print buffer -->
- [x] #50523 <!-- Avoid generic call in most cases for getproperty -->
- [x] #50635 <!-- `versioninfo()`: include build info and unofficial
warning -->
- [x] #50670 <!-- Make reinterpret specialize fully. -->
- [x] #50666 <!-- include `--pkgimage=no` caches for stdlibs -->
- [x] #50765 
- [x] #50764
- [x] #50768
- [x] #50767
- [x] #50618 <!-- inference: continue const-prop' when concrete-eval
returns non-inlineable -->
- [x] #50689 <!-- Attach `tanpi` docstring to method -->
- [x] #50671 <!-- Fix rdiv of complex lhs by real factorizations -->
- [x] #50598 <!-- only limit types in stack traces in the REPL -->
- [x] #50766 <!-- Don't partition alwaysinline functions -->
- [x] #50771 <!-- re-allow non-string values in ENV `get!` -->
- [x] #50682 <!-- Add fallback if we have make a weird GC decision. -->
- [x] #50781 <!-- fix `bit_map!` with aliasing -->
- [x] #50172 <!-- print feature flags used for matching pkgimage -->
- [x] #50844 <!-- Bump OpenBLAS binaries to use the new GEMM
multithreading threshold -->
- [x] #50826 <!-- Update dependency builds -->
- [x] #50845 <!-- fix #50438, use default pool for at-threads -->
- [x] #50568 <!-- `Array(::AbstractRange)` should return an `Array` -->
- [x] #50655 <!-- fix hashing regression. -->
- [x] #50779 <!-- Minor refactor to image generation -->
- [x] #50791 <!-- Make symbols internal in jl_create_native, and only
externalize them when partitioning -->
- [x] #50724 <!-- Merge opaque closure modules with the rest of the
workqueue -->
- [x] #50738 <!-- Add alignment to constant globals -->
- [x] #50871 <!-- macOS: Don't inspect dead threadtls during exception
handling. -->

Need manual backport:

Contains multiple commits, manual intervention needed:

Non-merged PRs with backport label:
- [ ] #50850 <!-- Remove weird Rational dispatch and add pi functions to
list -->
- [ ] #50823 <!-- Make ranges more robust with unsigned indexes. -->
- [ ] #50809 <!-- Limit type-printing in MethodError -->
- [ ] #50663 <!-- Fix Expr(:loopinfo) codegen -->
- [ ] #50594 <!-- Disallow non-index Integer types in isassigned -->
- [ ] #50385 <!-- Precompile pidlocks: add to NEWS and docs -->
- [ ] #49805 <!-- Limit TimeType subtraction to AbstractDateTime -->
@KristofferC KristofferC removed the backport 1.10 Change should be backported to the 1.10 release label Aug 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:codegen Generation of LLVM IR and native code compiler:llvm For issues that relate to LLVM
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants